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This paper looks at using descriptions of subject matter 
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performance on the National Assessment of Educational Progress (NAEP) . These 
descriptions of content, called achievement level descriptions (ALDs) , were 
initially conceptualized as exemplary statements of the knowledge and skills 
students should have at the various levels. Gradually, over the past decade 
the descriptions have evolved into comprehensive, rather than exemplary, 
statements of assessment content that must reflect the subject matter item 
pools to ensure consistency. This paper traces the policy development of the 
ALDs, discusses the various approaches to developing the ALDs over the past 
10 years, and reviews the form and function of the ALDs in the current NAEP 
context. The paper suggests that ALDs should be retained and that they play 
an important role in shaping the end results from the NAEP and providing 
replicability in the process. The ALD process seems to be working; however, 
the ALDs may not serve the reporting function very well. Several approaches 
could remedy this weakness to help make NAEP results understandable for the 
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Setting Student Performance Standards: 
the Role of Achievement Level Descriptions in 
the Standard Setting Process 



Mary Lyn Bourque 

National Assessment Governing Board 



Introduction 

This paper, based on ten years of experience with the National Assessment of 
Educational Process (NAEP), takes a closer look at the role of using descriptions of subject 
matter content to assist in the development and interpretation of student performance on 
the National Assessment. Initially, these descriptions of content, called achievement level 
descriptions (ALDs) were conceptualized as exemplary statements of the knowledge and skills 
that students should know and be able to do at the various levels (NAGB, 1990). Such 
statements were also used in earlier NAEP reports to interpret the cut scores (student 
performance standards) on the NAEP scale. The standards, called achievement levels, 
consist of three levels, Basic, Proficient, and Advanced. In reporting student performance on 
NAEP, the descriptions are exemplary statements of what students at a particular level are 
likely to know and be able to do. This distinction between what students ‘should know’ and what 
students demonstrate they ‘do know’ on the NAEP assessment led to some controversy 
over the precise meaning of the ALDs. Gradually over the past decade the descriptions have 
evolved into comprehensive rather than exemplary statements of assessment content that must 
reflect the subject matter item pools to ensure consistency between the two. 

This paper will trace the policy development of the ALDs, discuss the various 
approaches to developing the ALDs over the past decade, and review the form and function 
of the ALDs (and various derivations) in the current NAEP context. 

Evolution of NAEP Policy Definitions of the Standards 

Lu rkin g behind the ALDs are something called policy definitions, statements of the 
evidence of student achievement for each level that are independent of grade and subject 
matter content. The policy definitions, originally formulated by the National Assessment 
Governing Board (NAGB, 1990), were designed to lay out broad expectations for students 
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performance, for example, Basic signaled partial mastery , Proficient was competency oner challenging 
subject matter. The initial policy statements were elaborate (in the absence of ALDs they 
needed to be), and worse yet, they included statements of prediction, none of which could 
be substantiated by cross-sectional data. For example, some of the initial policy definitions 
included statements like “[students at the Proficient level] are well prepared for the next 
level of schooling.” Further, the initial ALDs were not well-balanced with respect the 
amount of detail, having far more at grade 12, and substantially less at grades 4 and 8. 

These initial policy statements were sorely criticized by various groups, including the 
National Academy of Education, the NAEP Technical Review Panel, and other well-known 
researchers and evaluators (Burstein et al, 1996; Koretz & Deibert, 1995; Linn et al, 1991; 
NAE, 1992; NAE, 1993a; NAE, 1993b; NAE, 1996; NAE, 1997). Consequently, in 1993 
they were revised. The newer versions were streamlined having no predictive statements, 
were fully balanced applying to all grades and subjects equally well, and tapped into the 
cognitive processes related to the levels. However, they are quite demanding in their 
expectations of what students should know and be able to do. See Figure 1 for a summary 
of the current statements. 



Insert Figure 1 about here 

There are some that claim that the policy definitions drive the high standards on 
NAEP. They argue this because it is the policy definitions that become operationalized into 
the ALDs. Defining Proficient as “competency over challenging subject matter” drives the 
ALDs into the higher ranges of subject mater content and thus the higher ranges of mastery. 



Evolution of NAEP Achievement Level Descriptions (ALDs) 

The use of ALDs in standard setting in general was not common in 1990. As a 
matter of fact in the 1990 NAEP initiative, there were none (Hambleton & Bourque, 1991). 
Panelists were required to translate policy definitions direedy into cut scores on the NAEP 
scale without benefit of the intermediary steps of using grade- and subject matter- 
appropriate descriptions of content. Subsequent to developing the cut scores, descriptions 
were derived for reporting purposes by examining the items that anchored on the scale at or 
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around the cut score, and gener alizin g to statements of content for the grade and 
achievement levels. These descriptions were akin to the anchor descriptions developed for 
NAEP reports in the mid-to-late 1980s. 

However, starting with the 1992 standard setting, based on lessons learned from 
1990, the use of ALDs with the standard setting panels became standard operating 
procedure. The timing of the development of ALDs has varied from cycle to cycle. Figure 
2 displays the relationship between early and current policy definitions, the NAEP cycle 
involved, and how and when the ALDs were developed for that specific cycle. Starting in 
1992 NAEP has always used ALDs in developing the standards. However, for mathematics 
and reading (1992), they were developed by the standard setting panels at the beginning of 
the process, and refined throughout the process (ACT, 1993a; ACT, 1993b). This was not 
the ideal situation since it took several days to train panelists to the point where they could 
craft statements based on the assessment framework and the item pools. Thus, it took time 
away from the real task of developing the cut scores and selecting exemplar items. On the 
other hand, having the panelists develop the ALDs gave them a “buy-in” to the whole 
process, as well as an understanding of the framework and item pools that was valuable as 
they worked through the rating process for setting the levels. 

Insert Figure 2 about here 

However, NAGB felt that the wrong chronology was being used. In other words, 
the ‘standards’ (namely, ALDs) should be set first, and then cut scores developed to reflect 
the content standards. So starting in 1994, preliminary achievement level descriptions 
(P ALD s) were crafted by the consensus panels that develop the assessment framework. 

Who better knew the content and what students should know and be able to dp than the panels 
who were developing the content of what was going to be assessed? The PALDs were 
pr eliminar y because they would serve to delimit the domain and the assessment content, and 
act as guides for the item writers. In addition, it was likely that some content identified in 
the PALDs might not be included in the final selection of items on the assessment. 
Therefore, having preliminary descriptions provided some flexibility later on. 

In the U.S. history and world geography assessments (1994) the PALDs were given 
to the achievement level setting panels to finalize during the standard setting process (ACT, 
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1994a; 1994b). This procedure still required enormous amounts of time and training. 

Panelists needed to fully understand the assessment frameworks and have good familiarity 
with the item pools before attempting any modifications of the PALDs. Consequently, even 
though this approach was an improvement over previous NAEP cycles, it was not the final 
answer. 

Therefore, in civics and writing (1998) yet a third procedure was devised to move 
from policy definitions to preliminaiy achievement levels descriptions to final achievement 
level descriptions (ACT, 1997a). The 1998 process simply removed the task of developing 
the A TP s from the standard setting panels altogether. Prior to the level setting in 1998, 
content panels were brought together to examine the public comments from a broad review 
of the PALDs. The review was conducted via the Internet and focus groups. The content 
experts took the comments into account and revised the statements for clarity, accuracy, and 
media appeal. These finalized ALDs then became Qians to the standard setting process, 
much like the item pool and the frameworks are Quens. We had anticipated a lack of 
commitment to the ALDs simply because the panelists had no part in their development. 
However, out fears proved unsubstantiated in the long run. 

I will comment only briefly on the 1996 science process for the sake of 
completeness. The ori ginal ALDs were developed identically to the 1994 process, with 
P ALD s coming out of the framework consensus group, and ALDs developed by the 
standard setting panels (ACT, 1997d). However, in the end, the cut scores developed by the 
panels were rejected by NAGB. Therefore, the ALDs and the NAGB-selected cut scores 
did not reflect each other and could not be used in reporting the results of the 1996 NAEP 
science assessment. Subsequently, new sets of ALDs were developed to reflect the NAGB 
cut scores using a scale anchoring procedure. 

There is one more variation of these descriptions that plays a role in the level setting 
process, namely, Borderline Descriptions (BDs). The PALDs and ALDs always describe 
what students should know and be able to do whose performance is in the range being described. 
So, for example, the Basic achievement level description outlines the content expectations for 
students whose performance is in the range from the Basic cut score on the scale up to the 
Proficient cut score. Similarly, the Proficient achievement level description outlines the 
content expectations for students whose performance is in the Proficient range, i.e., from the 
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Proficient cut score up to the Advanced cut score. Figure 3 displays the relationship 
between the ALDs and BDs on the NAEP scale. 

Insert Figure 3 about here 

However, in training panelists to rate items and set cut scores, it is very necessary for 
them to think about the borderline performance of examinees, that is, what is it that students 
just need to know to move from one level up to the next? Obviously, if the ALDs represent 
a range of performance, the borderline performance will be a subset of that full range. 

These are what are called Borderline Descriptions (BDs) in NAEP. Borderline descriptions have 
been used in the NAEP process since the 1996 science effort. 

In the NAEP process the BDs are developed by the standard setting panelists after 
having extensive training and developed an understanding of the assessment, the framework, 
the item pools, and most especially, the achievement level descriptions. The BDs are 
developed in grade groups and are used by the panelists in providing their item ratings. 

The Form and Function of Achievement Level Descriptions 

In the earlier sections of this paper the author has reviewed several iterations of 
statements of content that are important in the level setting process. There are policy 
definitions, preliminary descriptions, final descriptions, and borderline descriptions. All of 
this played out in the role of the ALDs in the various subject area standards development. 
Throughout this evolution, the descriptions have played a variable but critical role in 
developing and validating the levels. Each set of descriptions has a form and one or more 
functions in the process. The remainder of this paper will focus on the ALDs and detail the 
relationship between form and function. 

The current ALDs are dense statements of content expectations. They have a heavy 
curricular bias in their language, are frequently replete with jargonese, and represent 
generalizations from the assessment framework and the item pools. That is their form, and 
this characterization is especially tme of the early ALDs, including reading, mathematics, 

U.S. history, world geography, and science. It is somewhat less tme of the current ones in 
civics and writing. In developing the recent sets of descriptions attempts were made to have 
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them reviewed by media experts who improved the public appeal of the language and made 
them more generally accessible to the NAEP readership. 

What is the function of the ALDs? Their most important function is to provide a 
mental framework or structure for standard setting panelists. It is critical for all panelists to 
have a common understanding of the meaning of Basic, Proficient, and Advanced. Since the 
ALDs are derived to reflect the NAGB policy definitions, it is essential that during the 
process that panelists keep focused on the policy levels established by NAGB. The ALDs 
assist in achieving that focus. See Figure 4 for an example of the grade 8 mathematics 
ALDs. 



Insert Figure 4 about here 

Further, other AID functions include assisting panelists in rating the items in the 
item pool. The ALDs assist in aligning the expected content with the ratings assigned to 
items by panelists. In addition, since all panelists are using a common filter or screen 
through which to view the items, achieving consistency from level to level, and, in some 
cases, across grades, becomes a possible goal. Finally, the ALDs assist panelists to 
internalize the meaning of borderline perfimnance in the standard setting process. 

These are the current functions of the ALDs, but are there additional desired 
functions? The answer is a resounding yes. The NAEP achievement levels are reported as a 
package, that is, there are verbal descriptions of content (ALDs), there are exemplar 
exercises taken from the assessment itself, and there are the cut scores. In addition, the 
NAEP reports provide the percentages of students in the nation and the states who have 
attained the Basic, Proficient, and Advancedlevels. 

However, for too many NAEP reports, the ALDs have come up short in terms of 
reporting to the public on NAEP results. The curricular language of the current form is not 
understood well by those outside the curriculum area. The ALDs do not help to interpret 
the NAEP performance levels (cut scores) to the American public, nor do they provide the 
public press and media outlets with a clear understanding of what it means to perform at the 
Basic, Proficient, or Advanced levels. There is, for example, no known case where the media 
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have printed the ALDs in part or in whole as part of the press release on NAEP results. 1 
The ALDs were originally intended to serve as a supplement to the exemplar exercises in 
explaining what students know and can do in NAEP. It is the author’s view that they have 
failed in this regard. 

Summary and Conclusions 

Where do we go from here? Should NAEP abandon the use of the ALDs and other 
variations thereof? It is the author’s contention that the ALDs as such should be retained in 
the process. They play such a critical role in helping to define the standards that to drop 
them would be to place the procedural validity of the entire process at risk. Although we 
have no firm empirical evidence in this area, the data that are available from pilot studies and 
operational settings would suggest that the ALDs play a vital role in shaping the end results 
and provi din g replicability in the process. In several subject areas we have seen very similar 
results between the pilot cut scores and the operational cut scores, suggesting that such 
consistency did not happen by accident (see for example the results in the 1996 science, 

ACT 1997b; ACT, 1997c; and ACT 1997d). 

After a decade of trying, the process seems to be about right (see Figure 5). NAGB 
sets the policy definitions up front. These in turn are given to the framework consensus 
panels to operationalize in terms of a specific grade level and content area on a preliminary 
basis. The preliminary statements are then widely reviewed by a variety of NAEP audiences, 
inclu din g other content specialists, stakeholders (e.g., SEAs and LEAs), NAEP users (e.g., 
teachers, administrators, policymakers, parents, etc.), and others. The reviews are 
incorporated into a revised version which NAGB approves before the standard setting 
panels use them to develop their recommendations on cut scores and exemplars. Finally, the 
whole package, i.e., cut scores, descriptions, and exemplars, goes back to the policymaking 
body, NAGB, for approval and adoption. 



1 A review of over 400 press clippings from 1990 through 1999 of NAEP results has shown no releases where 
the ALDs were re-printed by the media. 
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Insert Figure 5 about here 



That being said, however, still leaves the plaguing question of whether or not the 
ALDs serve the reporting function very well. It is this author’s contention that they do not. 
The appropriate solution to this dilemma is not clear. There are a number of approaches 
that could be used. One approach is to “tinker” with ALDs and re-write them for reporting 
purposes. The author strenuously disagrees with this approach. First, the re-writes are 
subject to the vagaries of language, and may or may not reflect faithfully the ALDs adopted 
by NAGB. Second, if the ALDs are “official” descriptions of the standards, what are the re- 
writes, “unofficial?” How will the public for whom the reports are intended untangle the 
official interpretation from the unofficial? Third, there are three descriptions for each grade, 
and nine across the subject area. Re-writes would lead to 18 descriptions, an unwieldy count 
by any standards. 

Another approach is to capitalize on the exemplars for helping to describe the kinds 
of knowledge and skills that examinees are likely to know and be able to do on NAEP. The 
reporting function of the descriptions is to assist in the interpretation of NAEP 
performance. The exemplars seem to do this quite well. It might be possible to extract 
from the exemplar exercises the knowledge and skills students must have in order to answer 
the exemplar correctly. This would be a fairly focused statement of knowledge and skills, 
deriving solely from a single item, but would help to clarify the interpretation. Such 
exemplar-derived statements would not be as comprehensive as are the ALDs, nor would 
they be generalized statements as were the scale anchor descriptions. But if examinees are 
indeed answering the exemplars correctly, then such statements do assist in interpreting the 
NAEP performance. See Figure 6 for an example of this approach. 

Insert Figure 6 about here 

Another approach to helping the reader of NAEP reports to interpret the results is 
to simply provide them a collection of exemplars that are likely to be answered correcdy by 
examinees whose performance is in the Basic range or the Proficient range. In the standard 
setting process we provide such lists of exemplars to panelists for them to select the most 
appropriate exemplars (space limitations preclude printing a full display of all exemplars in 
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NAEP reports usually). Providing the reader with the whole list could be quite helpful. 
Figure 7 is an example this approach, and was used in the 1996 NAEP science report 
produced by NAGB. 



Insert Figure 7 about here 

It is the conclusion of this author that much more work needs to be done in the area 
of using the ALD s as verbal interpretations of the NAEP results. In attempting to answer 
the question, “What do fourth graders know who are performing at the Proficient level?” 
there are three (or four depending on how you count) approaches one might take. One is 
numbers. Scale scores and cut scores on the NAEP scale just do not have any intuitive 
me anin g to the American public. They seldom are reported by the public press as a means 
of “telling the stoiy” about student performance. A second approach related closely to 
numbers is graphical displays. These seem to work - sometimes -- depending on the level of 
sophistication of the display and the reader. The media to report comparative state-by-state 
performance on NAEP occasionally uses graphic displays. The third approach is test items. 
These seem to work best, but are in limited supply, and are never re-printed in their entirety 
by the media. The fourth approach is words, and this returns us to the ALDs. 

My final advice is proverbial - don’t throw the baby out with the bath water. NAEP 
should keep the ALDs since they function well in the standard setting process and are a 
critical part of that process. However, NAEP needs to work hard on fixing the reporting 
and interpretation of NAEP scales problem. NAEP is for the American public and, at one 
level, should be as understandable as your hometown newspaper, the Dow Jones Industrial 
average, or the Consumer Price Index. This lack of understanding is not a new problem 
created by the use of standards in reporting the NAEP results. However, the audiences for 
NAEP reports have changed and expanded in recent years. This is cause for celebration and 
concern. More research needs to focus on how best to get the message out to NAEP’s new 
and expanded audiences. 
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This level signifies superior performance 



Figure 2 



Relationship of Early and Current PDs and the Evolution of ALDs, 

1990-1998 NAEP cycles 



Policy Definitions 
(PDs) 



Early PDs (1990-1992) 

• Elaborate 

• Statements of 
prediction 

• Not balanced 

• Criticized by NAE, 
NAEP-TRP, RAND, etc. 



Current PDs (1994-1998) 

• Streamlined 

• No predictive 
statements 

• Fully balanced 

• Tap into cognitive 
processes 

• Hard to understand by 
general public 

• Linked to language in 
Goals 2000 legislation 

• Linked to Title 
Legislation 

• Model for state 
assessments 

• Model for some 
commercial tests 



Achievement Level 
Descriptions (ALDs) 



Math, 1990 «*=$> 



Math, 1992 
Reading, 1992 



US History, 1994 

World Geography, 1994 



Science, 1996 



Civics, 1998 
Writing, 1998 



How/When ALDs 
Developed 



• After standard setting 

• For reporting purpose 

• During process by 
panelists 

• Revised by content 
specialists later 



• PALDs by consensus 
panels 

• ALDs finalized by AL 
panelists 



• PALDs by consensus 
panels 

• ALDs finalized by AL 
panels 

• Reporting ALDs 
developed by an 
anchoring panel to 
correspond to Bd- 
selected outpoints 



• PALDs by consensus 
panels 

• ALDs by content panels 
(became a given in the 
AL process) 
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Figure 3 

Achievement Level Descriptions and the 

NAEP scale 
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Description of Mathematics Achievement Levels for 
Basic, Advanced and Proficient Eighth Graders 

The five NAEP content areas are (1) numbers and operations, (2) measurement, (3) 
geometry, (4) data analysis, statistics, and probability, and (5) algebra and functions. Skills are 
cumulative across levels — from Basic to Proficient to Advanced. 



Eight-grade students performing at the basic level should exhibit evidence of 
conceptual and procedural understanding in the five NAEP content areas. This 
level of performance signifies an understanding of arithmetic operations - 
including estimation - on whole numbers, decimal, fraction, and percents. 




Eighth graders performing at the basic level should complete problems correctly with the 
help of structural prompts such as diagrams, charts, and graphs. They should be able to solve 
problems in all NAEP content areas through the appropriate selection and use of strategies and 
technological tools - including calculators, computers, and geometric shapes. Students at this 
level also should be able to use fundamental algebraic and informal geometric concepts in 
problem solving. 

As they approach the proficient level, students at the basic level should be able to 
determine which of available data are necessary and sufficient for correct solutions and use them 
in problem solving. However, these 8 th graders show limited skill in communicating 
mathematically. 



Proficient 294 Eight-grade students performing at the proficient level should apply 
mathematical concepts and procedures consistently to complex problems in 
the five NAEP content areas. 



Eight graders performing at the proficient level should be able to conjecture, defend their 
ideas, and give supporting examples. They should understand the connections between 
fractions, percents, decimals, and other mathematical topics such as algebra and functions. 
Students at this level are expected to have a thorough understanding of basic level arithmetic 
operations — an understanding sufficient for problem solving in practical situations. 

Quantity and spatial relationship in problem solving and reasoning should be familiar to 
them, and they should be able to convey underlying reasoning skills beyond the level of 
arithmetic. They should be able to compare and contrast mathematical ideas and generate their 
own examples. These students should make inferences from data and graphs; apply properties 
of informal geometry; and accurately use the tools of technology. Students at this level should 
understand the process of gathering and organizing data and be able to calculate, evaluate, and 
communicate results within the domain of statistics and probability. 



Eight-grade students performing at the advanced level, should be able to 
reach beyond the recognition, identification, and application of mathematical 
rules in order to generalize and synthesize concepts and principles in the five 
NAEP content areas. 




Eighth graders performing at the advanced level should be able to probe examples and 
counterexamples in order to shape generalizations from which they can develop models. Eighth 
graders performing at the advanced level should us number sense and geometric awareness to 
consider the reasonableness of an answer. They are expected to use abstract thinking to create 
unique problem-solving techniques and explain the reasoning process underlying their 
conclusions. 
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Figure5 

Relationship of Policy Definitions (PDs), PALDs, 
ALDs, BALDs in Current NAEP Standard Setting 
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Figure 6 



Grade 12 Science Exemplars + 



BASIC 



1 . Some students wee studying water in the environment. They filled one 
sample jar with ocean water and another sample jar with fresh water 
from the lake. The labels on the jars fell off, and the water in both jars 
looked the same. Describe a test, other than tasting or smelling the 
water, that the students could do to determine which jar held the lake 
water. Explain how the test would work. 

6quU lesi {kzuAf&r jvr_ leHiui. U<£ up 



This question measures the students’ ability to plan a scientific test and to explain how the test 
would work. Students at the Basic level are likely to provide a method for the experiment, while 
the students at the Proficient level also provide its results. 



Proficient 



2. Coal is burned in a power plant that produces electricity. In a house 
miles away, a lightbulb is turned on. Describe the energy transformations 
involved. 



CoaI beluk it'J&s off toktek jjveg 



Compare the amount of energy released in one hour by burning the coal, 
the amount of energy received from the power plant in one hour by the 
house, and the amount of light energy produced in one hour by the 
lightbulb. Explain any differences among these three amounts of energy. 



Yltg aw punt c4 euAriij released [in] t kmr Kj coA Is wuA wore Ikw \V& c4 

tvAf^ fueled jrow \V& pow&rpUnI In ova hour llc*n Kj tits kcuse toktek Is more titg mmouiA 
lljjtl I [in] om kgu<~ \>q 1kg lljklMl. All fkls &aaV\\wa Is |?ASSgs, losgs 

gHgrj^. 



Students at the Advanced level demonstrate a full understanding of energy transformations in 
technological systems and can explain differences among the energy transformations. Students 
at the Proficient level are likely to only provide one portion of the full response 
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What Twelfth Graders Know and Are Able To Do In Science 
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